98¢ /MFlop, Ultra-Large-Scale Neural-Network Training on a PIII Cluster

Artificial neural networks with millions of adjustable parameters and a similar number of training examples are a potential solution for difficult, large-scale pattern recognition problems in areas such as speech and face recognition, classification of large volumes of web data, and finance. The bottleneck is that neural network training involves iterative gradient descent and is extremely computationally intensive. In this paper we present a technique for distributed training of Ultra Large Scale Neural Networks1 (ULSNN) on Bunyip, a Linux-based cluster of 196 Pentium III processors. To illustrate ULSNN training we describe a preliminary experiment in which a neural network with 1.8 million adjustable parameters is being trained to recognize machine-printed Japanese characters from a database containing 6 million training patterns. The simulation is still underway, with an average performance during the first 56 hours of operation (the elapsed time in the simulation prior to this paper’s submission) of 152 GFlops (single precision). With a machine cost of $149,500, this yields a price/performace ratio of 98¢/MFlop (single precision). For comparison purposes, training using double precision and the ATLAS DGEMM produces a sustained performance of 70 MFlops or $2.13 / MFlop (double precision).